AdaDelay: Delay Adaptive Distributed Stochastic Optimization

We develop distributed stochastic convex op-timization algorithms under a delayed gradi-ent model in which server nodes update pa-rameters and worker nodes compute stochas-tic (sub)gradients. Our setup is motivated bythe behavior of real-world distributed com-putation systems; in particular, we analyzea setting wherein worker nodes can be dif-ferently slow at different times. In contrastto existing approaches, we do not impose aworst-case bound on the delays experiencedbut rather allow the updates to be sensitiveto the actual delays experienced. This sen-sitivity allows use of larger stepsizes, whichcan help speed up initial convergence with-out having to wait too long for slower ma-chines; the global convergence rate is stillpreserved. We experiment with different de-lay patterns, and obtain noticeable improve-ments for large-scale real datasets with bil-lions of examples and features.